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Abstract 


This  study  aims  to  design,  develop,  and  test  a  portable,  light\A/eight,  noninvasive  neuroimaging 
system  that  supports  simultaneous  electroencephalogram  (EEG)  and  functional  NIR 
spectroscopic  (fNIRS)  acquisition  for  biological  or  cognitive  neuroscience  studies  in  operational 
environments.  The  system  features  novel  EEG/NIRS  electrodes,  kno\A/n  as  electro-opodes,  and 
miniaturized  supporting  hardware/software.  In  the  past  few  years,  our  team,  composed  of 
faculty,  postdoctoral  fellows  and  graduate  students,  has  designed  and  developed  dry  EEG  and 
fNIR  sensors  that  allow  non-invasive  and  non-intrusive  acquisition  of  EEG  and  fNIR  signals.  We 
have  also  designed  the  form  factor  of  Electro-optodes  that  integrates  the  EEG  and  fNIR  sensors 
in  an  elastic  cap.  We  further  evaluated  the  quality  of  EEG  signals  acquired  by  the  dry 
spring-loaded  EEG  sensors  and  fNIR  sensors  through  three  experiments.  This  study  has  also 
made  a  lot  of  progress  in  designing  and  developing  a  data-acquisition  VLSI  chip  that  can 
acquire,  amplify,  digitize  and  process  EEG  and  NIRS  data.  Lastly,  to  improve  the  fidelity  of  the 
EEG  recordings,  we  have  developed  and  implemented  real-time  artifact  correction  algorithms. 
Artifact  Subspace  Reconstruction,  for  online  and  real-time  rejection  of  artifacts  that  often 
contaminate  EEG  signals  recorded  in  real-world  environments. 


1 


Objective 

This  study  aims  to  design,  fabricate,  and  test  a  truly  portable,  lightweight,  noninvasive, 
neuroimaging  system  that  supports  simultaneous  electroencephalogram  (EEG)  and  functional 
Near  Infrared  Spectroscopic  (fNIRS)  acquisition  to  provide  complementary  information  of  the 
brain.  The  acquired  neural  dynamics  could  be  comparable  to  those  obtained  from  simultaneous 
EEG  and  functional  Magnetic  Resonance  Imaging  (fMRI)  recordings.  The  envisioned 
dual-modality  neuroimaging  system  will  feature  our  newly  developed  EEG/NIRS  electrodes, 
known  as  electro-optodes,  based  on  Micro-Electro-Mechanical  System  (MEMS)  technology  and 
miniaturized  supporting  hardware  and  software  for  biological  or  cognitive  monitoring  of 
participants  in  operational  environments.  Another  major  goal  of  this  project  is  to  develop 
advanced  signal-processing  methods  and  software  for  (1)  increasing  signal-to-noise  ratio  of  the 
acquired  EEG  and  optical  imaging,  (2)  2-D  and  3-D  image  reconstruction  and  rendering  to 
reconstruct  the  optical  images  for  investigating  neurovascular  coupling  with  minimum 
inter-modality  interferences  and  preparation  demands. 


Scientific  Barriers 

The  advantages  of  simultaneous  EEG/fMRI  recordings  have  been  explored  by  many  studies  and 
received  increasing  attention  in  neuroscience  community  [Ives  et  al.  (1993);  Huang-Hellinger  et 
al.  1995;  Lemieux  et  al.,  1997].  However,  simultaneous  EEG/fMRI  recordings  have  formidable 
technical  problems.  For  example,  in  EEG/fMRI  recording,  any  ferromagnetism  in  the  scanner 
can  be  dangerous  for  the  subject,  and  will  certainly  cause  unacceptable  loss  in  the  BOLD  image. 
Then,  small  movements  of  the  electrodes  or  cables  in  a  strong  magnetic  field  will  introduce 
artifactual  currents.  The  recorded  EEG  signals  will  be  completely  useless  before  the  MR  artifacts 
can  be  removed.  To  remove  the  MR-induced  artifacts,  the  onset  timings  and  the  waveforms  of 
the  MR  artifacts  need  to  be  precisely  registered  during  the  acquisition  of  each  MR  image  slice. 
This,  in  turn,  requires  high  sampling  rate  for  EEG  recordings  (at  least  5  KHz  compared  with  250 
Hz  for  regular  EEG  recordings).  On  the  other  hand,  EEG  electrodes  may  also  introduce 
deformation  of  the  magnetic  field  of  the  MR  scanner  and  in  turn  cause  signal  void  in  (f)MR 
images  around  the  scalp  electrodes.  All  these  interferences  make  simultaneous  EEG/fMRI  and 
post-processing  extremely  difficult.  Therefore,  a  truly  small,  lightweight,  battery-powered 
EEG/fMRI  system  for  concurrent  EEG/fMRI  recordings  of  the  freely-moving  subjects  in 
real-world  environments  will  be  extremely  difficult,  if  ever  possible,  in  the  near  future. 
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Significance 

This  study  is  integrating  a  novel  EEG/fNIRS  electro-optodes  that  can  be  used  as  an  EEG 
electrode  and  a  NIR  light  emitter  and  photo  detector,  supporting  \A/earable  data  acquisition 
(DAQ),  signal-processing  and  image  rendering  software  into  a  wearable  neuroimaging  system. 
We  will  also  develop  advanced  2-D  and  3-D  image  reconstruction  algorithms  to  reconstruct  the 
optical  images  from  the  recorded  fNIRS  signals.  The  envisioned  simultaneous  EEG/fNIRS 
acquisition  system  is  completely  novel  and  different  from  any  other  currently  available  systems. 
First,  the  envisioned  system  features  brand  new  dry  EEG/NIRS  electro-optodes,  which  can 
function  as  an  NIR  emitter,  detector,  and  an  EEG  electrode  without  requiring  any  skin 
preparation  and  conductive  paste.  Most  importantly,  the  electro-optodes  also  comprises 
built-in  NIR  light  sources  and  NIR  waveguide  to  bypass  the  high  reflectance  of  the  human 
dead-skin  layer  such  that  low-power  NIR  light  sources  will  be  sufficient  for  high-quality 
recording.  In  addition  to  the  innovative  hardware  design,  fabrication,  and  integration, 
ICA-based  signal  processing  can  separate  the  mixture  signals  caused  by  multiple  light  sources 
and  multiple  possible  pathways.  As  a  result,  we  are  able  to  reconstruct  a  3-D  optical 
tomography,  which  may  be  more  specific  to  the  underlying  BOLD  effects.  In  turn,  we  should  be 
able  to  better  assess  the  inter-relationship  between  the  BOLD  effects  and  the  neural  activities 
measured  by  the  simultaneously  recorded  EEG  signals. 

Why  is  this  research  of  value  to  the  Army? 

Given  the  increased  operation  tempo  and  complexity  that  soldiers  are  facing  today  and  in  the 
future,  new  approaches  and  technologies  will  be  needed  to  maximize  and  maintain  attentional 
capacity.  Understanding  how  this  complexity  affects  human  sensory,  perception,  and  cognitive 
performance  to  effectively  exploit  the  cognitive  capabilities  of  individuals  is  thus  very  crucial  to 
improve  and  optimize  individual  or  team  performance  in  military  environments.  Monitoring  the 
neurophysiological  activity  of  Soldiers  within  a  complex  environment,  however,  poses  severe 
measurement  challenges  to  the  current  laboratory-oriented  sensing  technology.  Successful 
development  of  a  multi-modality  neuroimaging  system  could  allow  assessment  of  individual 
neurocognitive  performance  in  military  environments  and  could  also  play  important  roles  in 
enhancing  individual  and  team  learning  during  military  training. 
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Accomplishments 


During  the  past  few  years,  we  have  (1)  combined  dry  EEG  electrode  and  NIRS  probe  into  one 
integrated  electro-optode  for  simultaneously  EEG  and  fNIRS  recording;  (2)  developed  a  VLSI 
neuroimaging  chip  for  data  acquisition;  (3)  conducted  three  experiments  to  validate  the 
feasibility  of  the  custom-made  fNIR  probe;  and  (4)  developed  online  automatic  artifact  removal 
approach  to  improve  the  fidelity  of  the  EEG  recordings.  The  details  of  the  achievements  are 
itemized  below: 

1.  Integrated  Dry  EEG  Electrode  and  NIRS  Probe  Into  an  Electro-optode  for  simultaneous  EEG 
and  fNIR  recording 

The  main  purpose  of  this  project  is  to  develop  an  electro-optode  that  combines  the  capability 
of  simultaneously  recording  both  EEG  and  fNIRS  signals  from  the  same  site  by  integrating  both 
EEG  electrode  and  NIR  probe  into  one  electro-optode.  Figure  1  shows  the  design  of  the 
electro-optode  on  a  PCB  with  C-shaped  design  to  record  signals  from  heir-bearing  sites.  Such  a 
design  has  proved  capable  of  placing  the  IR/Red  LEDS  and  photodiodes  more  closely  to  the 
scalp  without  the  interference  from  the  hair.  We  have  also  conducted  experiments  to  test  the 
feasibility  of  such  an  integrated  electro-optode  by  examining  the  quality  of  the  acquired  fNIRS 
and  EEG  signals. 


EEG  Electrode  NIRS/EEG  Signal  Processing  1C 


IR/Red  LEDs 


Figure  1.  The  integration  of  dry  EEG  electrode  and  NIRS  probe  to  form  a 
electro-optode  for  simultaneous  EEG  and  fNIR  recordings. 

2,  A  VLSI  Neuroimaging  Chip 
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To  acquire  EEG  and  fNIR  data  measured  by  the  Electro-Optodes,  we  have  designed  and 
fabricated  a  single  neuroimaging  chip  featuring  an  analog  front-end,  analog-to-digital  converter 
(ADC)  and  digital  signal  processing  (DSP)  units.  The  architecture  of  the  neuroimaging  chip  is 
shown  in  Figure  2.  We  are  currently  testing  the  functionality  of  the  VLSI  chip. 


Figure  2,  The  architecture  of  a  neuroimaging  chip. 


The  specification  of  the  proposed  is  shown  in  Table  1. 

Table  1.  The  specification  of  the  rehabilitation  chip 


Item 

Specification 

CMOS  Technology 

0.18  urn 

Die  size 

2.5x2.5mm^ 

Operation  voltage 

1.8V 

Clock  rate 

66MHz 

Power  consumption 

96mW(Digital) 
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3.  Three  Experiments  to  Validate  the  Custom-made  NIRS  Probes 

Therefore,  we  had  conducted  a  series  of  experiments  to  test  the  feasibility  by  checking  not  only 
the  signal  quality  but  also  the  derived  hemodynamic  responses  from  the  recorded  signals. 
These  experiments  included  (1)  a  vascular  occlusion  test  on  the  forearm  regions  of  human 
subjects,  (2)  a  breath  holding  test  on  the  human  subjects,  and  (3)  an  fNIR  experiment  during  a 
mental  arithmetic  task.  The  detail  experimental  paradigms  and  the  results  are  itemized  below. 

3.1  A  Vascular  Occlusion  Test  on  the  Forearm  Regions  of  Human  Subjects 

To  verify  the  NIRS  system  and  the  algorithm,  we  assess  the  variations  of  the  concentration  of 
oxyhemoglobin  ([Hb02])  and  deoxyhemoglobin  ([Hb])  when  external  force  was  applied  to  the 
forearm  to  block  the  blood  flow.  Figure  3  shows  the  experimental  setup.  The  NIRS  probe 
including  a  PD  and  an  LED  was  placed  on  the  left  forearm  of  the  subject,  and  the  cuff  was 
normally  placed  smoothly  and  snugly  around  the  upper  arm.  In  the  experiment,  the  blood 
pressure  was  increased  to  220  mmHg  and  held  by  an  inflatable  cuff  of  a  sphygmomanometer, 
and  then  released  the  cuff  after  15  seconds.  The  concentration  of  [Hb02]  and  [Hb]  were 
compared  across  the  three  conditions  (i.e.  pre-occlusion,  occlusion,  and  post-occlusion)  to 
assess  the  functionality  of  the  proposed  NIRS  probe. 


Figure  3,  Experimental  setup  of  an  inflatable  cuff  and  the  NIRS  probe. 


Figure  4  shows  the  changes  of  [Hb]  and  [Hb02]  with  and  without  external  pressure  of  the 
inflatable  cuff.  The  red  bold  line  at  0  sec  represents  the  time  point  of  adding  external  pressure 
to  increase  and  hold  the  blood  pressure  at  220  mmHg.  The  bold  blue  line  shows  the  cuff 
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releases  15  seconds  later.  The  red  and  the  blue  traces  depict  the  concentrations  of  [Hb02]  and 
[Hb],  respectively.  Figure  4(a)  is  the  control  condition  \A/ithout  external  pressure.  Figure  4(b) 
shows  the  changes  with  pressured  and  unpressured  cuff.  Apparently,  when  the  blood  pressure 
increased,  the  [Hb02]  of  forearm  decreased  and  the  [Hb]  increased  considerably. 


1(10' 


(a) 


X  10* 


Figure  4,  (a)  changes  of  [Hb]  and  [Hb02]  without  external  pressure;  (b)  changes  of  [Hb]  and 

[Hb02]  with  external  pressure. 
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3.2  Breath-Holding  Experiment  on  the  Humans  Brains 

Having  validated  the  concentrations  of  [Hb]  and  [Hb02]  in  the  forearm  vascular  occlusion 
experiment,  we  conducted  a  breath-holding  experiment  to  further  verify  the  proposed  NIRS 
system  for  the  brain.  The  breath-holding  experiments  were  introduced  to  control  the 
concentration  of  oxygen  in  the  brain.  In  the  experiments,  the  NIRS  probes  were  placed  at  Fpl, 
as  shown  in  Figure  5. 


Figure  5,  The  NIRS  probe  was  placed  at  Fpl  in  the  breath-holding  experiments. 


Figure  6  shows  the  experimental  procedure.  The  first  30  seconds  is  a  pre-rest  (control)  stage. 
After  the  pre-rest  stage,  subjects  should  start  to  exhale  all  of  the  air  out  of  their  lung  in  2 
seconds  for  creating  the  condition  of  ischemia  and  then  hold  breath  for  the  next  28  seconds. 
After  the  breath-holding  stage,  the  subject  can  breathe  normally  for  30  seconds  until  the  next 
trial  begins.  Each  subject  repeated  the  experimental  for  10  times  in  one  session  to  derive  an 
averages  hemodynamic  response  of  the  breath-holding  experiment. 


Pre-rest  Breath-holding  Rest  Stop 


Next  trial 

Figure  6,  The  timeline  of  the  breath  holding  experiment. 


Four  subjects  (SI,  S2,  S3  and  S4)  participated  in  the  breath-holding  experiments,  each 
participated  in  two  sessions.  Figure  7  shows  the  empirical  results  of  [Hb]  and  [Hb02].  The  bold 
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red  line  represents  the  onset  of  breath-holding  after  30-second  pre-rest  stage.  The  results 
shows  that  the  [Hb02]  and  [Hb]  would  increase  and  decrease,  respectively.  This  phenomena 
may  be  because  there  is  a  protective  mechanism  to  deliver  oxygen  to  the  brain  when  the 
oxygen  concentration  is  insufficient,  and  then  the  [Hb02]  of  subjects  would  increase  for 
protection. 
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Figure  7.  The  results  of  breath-holding  experiments  from  4  subjects,  each  performed  two 

sessions. 

3.3  fNIR  Experiment  with  a  Mental  Arithmetic  Task 

Sections  2.1  and  2.2  show  the  changes  of  [Hb02]  and  [Hb]  in  a  large  area  for  more  critical 
conditions,  such  as  vascular  occlusion  and  breath  holding.  Such  critical  conditions  might  create 
more  noticeable  hemodynamic  changes.  To  evaluate  the  feasibility  of  the  proposed  NIRS 
system  for  assessing  brain  activity,  we  then  conducted  a  more  realistic  experiment,  as  we 
would  normally  do  during  a  cognitive  task.  This  study  tested  the  hemodynamic  responses  over 
the  forehead  regions  during  a  mental  arithmetic  task.  Figure  8  shows  the  test  procedure.  Each 
trial  started  with  a  30-second  pre-rest  stage,  then  the  subjects  performed  a  15-second  mental 
arithmetic  test  (two-digit  to  two-digit  addition),  followed  by  a  15-second  rest  stage  as  the 
control.  The  subjects  repeated  the  procedure  10  times  to  derive  the  averaged  hemodynamic 
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responses.  Each  subject  participated  in  two  sessions  on  separate  occasions  that  were  two 
weeks  apart. 


Pre-rest  Counting  Rest  Stop 


30s  15s  15s 


Next  trial 

Figure  8,  The  procedure  of  the  mental  arithmetic  experiments 


Figure  9  shows  the  averaged  changes  of  [Hb02]  and  [Hb]  across  10  trials  from  two  subjects.  The 
[Hb02]  increases  the  [Hb]  decreased  once  the  subjects  started  counting  indicating  the  subjects' 
brain  consumed  more  oxygen  to  support  the  mental  arithmetic  operations. 
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Figure  9,  The  result  of  the  mental  arithmetic  experiments 
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To  compare  with  a  commercially  available  NIRS  system,  we  also  conducted  the  same  cogntiitve 
experiment  on  the  same  subjects  using  the  Hitachi  ETG-4000  system.  Figure  10  shows  the 
comparison  results  between  our  NIRS  system  and  ETG-4000  system. 


(uMW 


(b) 

Figure  10,  The  results  of  the  mental  arithmetic  experiments.  Left  panels  were  recorded  by 
ETG-4000,  and  right  panels  were  recorded  by  the  proposed  NIRS  system. 
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4.  Development  of  signal-processing  methods  to  improve  the  fidelity  of  EEG 
recordings  and  extract  brain  activities  in  the  recordings 

4.1  On-line  recursive  ICA 

EEG  data  collected  by  dry  electrodes  and  NIR  sensors  are  subject  to  severe  motion,  ocular  and 
muscle  artifacts.  Methods  are  needed  to  remove  these  sources  of  EEG  contamination.  Over  the 
past  decade,  investigators  of  this  proposal  [Makeig  et  al.,  1996;  Jung  et  al.,  1998;  2000a;  2000b] 
have  developed  a  generally  applicable  method  for  isolating  and  removing  a  wide  variety  of  EEG 
artifacts  by  linear  decomposition  using  independent  component  analysis  (ICA)  [Bell  and 
Sejnowski,  1995].  Despite  such  successes,  more  development  and  testing  are  needed  to  make 
the  algorithms  feasible  for  robustly  and  continuously  correcting  artifacts  in  the  EEG  data  from 
unconstrained,  freely-moving  subjects  in  everyday  environments.  As  the  dry  EEG  sensors  will 
not  have  direct  connect  to  the  scalp,  the  locations  of  sensors  with  respect  to  the  scalp  will 
change  from  time  to  time,  ICA-based  artifact-correction  mechanism  thus  needs  to  be  able  to 
derive  the  varying  unmixing  matrix  accordingly.  In  other  words,  we  need  to  develop  an  on-line 
ICA  algorithm  or  algorithms  for  EEG  monitoring  in  real-world  environments.  During  the  past 
two  years,  we  have  developed  a  new  recursive  algorithm  for  incremental  estimation  of 
independent  components  from  on-line  data.  The  algorithm  offers  the  convergence  properties 
of  batch  ICA  with  incremental  updates  of  a  form  similar  to  natural  gradient  (NG)  on-line 
information  maximization  (Infomax).  We  found  significant  gains  in  convergence  rate  over 
on-line  natural  gradient  ICA  [Akhtar,  et  al.,  2012;  Hsu  et  al,  under  review]. 

4.2  Online  automatic  artifact  correction 

In  the  past  few  months,  we  further  developed  and  implemented  real-time  artifact  correction 
algorithms.  Artifact  Subspace  Reconstruction  (ASR),  for  online  and  real-time  rejection  of  EEG 
artifacts  that  occur  and  are  estimated  on  a  short  (1-min)  time  scale  and  are  contained  in  a 
low-dimensional  spatial  data  subspace  [Mullen  et  al.,  2013].  ASR  is  a  highly  effective  and  robust 
spatial  artifact  rejection  that  intuitively  finds  and  projects  high-amplitude  artifact  subspaces  out 
of  the  data.  The  method  can  be  feasibly  applied  in  near-real  time  to  data  containing  a  variety  of 
artifacts.  The  algorithm  is  built  upon  BCILAB  [Kothe  &  Makeig,  2013],  an  EEGLAB  plugin  for  the 
design,  prototyping,  testing,  experimentation  with,  and  evaluation  of  Brain-Computer 
Interfaces  (BCIs),  and  other  systems  in  the  same  computational  framework.  Figure  11  below 
shows  a  segment  of  EEG  data  contaminated  by  blink  and  muscle  artifacts,  before  and  after  ASR 
artifact  removal. 
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Figure  11.10  sec  of  EEG  data  following  ASR  data  cleaning  (blue  trace)  superimposed  on 
original  data  (red  trace).  Figure  is  copied  from  [Mullen  et  al.,  2013]. 
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6.  COLLABORATIONS  AND  LEVERAGED  FUNDING 

Leveraged  Funding.  Co-PIs  of  this  study  joined  a  Research  Consortium  which  won  a  five-year 
contract  from  Army  Research  Laboratory  (ARL)  Cognition  and  Neuroergonomics  (CAN)  program 
in  2010.  The  project  is  to  develop  and  demonstrate  basic  cognitive  neuroscience  principles 
guiding  development  of  improved  human-systems  interactions  in  complex  information 
environments.  Under  the  CAN  program,  a  Collaborative  Technology  Alliance  (CTA)  has  been 
formed  by  the  U.S.  ARL  and  our  Research  Consortium  to  explore  basic  scientific  knowledge  in 
neuroscience,  human  factors,  psychology,  and  engineering  to  enhance  our  understanding  of 
Soldier  brain  function  and  behavior  in  complex  operational  settings,  assessed  outside  the 
confines  of  standard  research  laboratories.  This  study  of  developing  an  innovative  EEG/fNIR 
system  and  the  new  CAN-CTA  project  are  complimentary  as  this  study  is  developing  new 
sensors  to  measure  simultaneous  EEG  and  fNIR  signals  of  wearers,  whereas  the  CAN-CTA 
provides  an  unparalleled  platform  for  us  to  test  and  explore  military  applications  of  our  new 
neuroimaging  modality. 

Collaborations.  We  recently  visited  Dr.  Harold  Szu  of  U.S.  Army  Night  Vision  &  Electronic 
Sensors  Directorate  (NVESD)  to  discuss  potential  applications  of  new  brain  imaging  modality. 


7.  CONCLUSIONS 

This  study  designed,  developed,  and  tested  a  portable,  lightweight,  noninvasive  neuroimaging 
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system  that  supported  simultaneous  EEG  and  functional  NIR  acquisition  for  biological  or 
cognitive  neuroscience  studies.  Over  the  past  few  years,  we  have  developed  novel  EEG/NIRS 
electro-opodes  that  allow  non-invasive  acquisition  of  EEG  and  fNIR  signals,  and  miniaturized 
supporting  hardware/software.  We  then  conducted  three  experiments  to  demonstrate  the 
feasibility  of  non-invasive  and  non-intrusive  acquisition  of  EEG  and  fNIR  signals.  We  have  also 
developed  and  implemented  real-time  artifact  correction  algorithms.  Artifact  Subspace 
Reconstruction,  for  online  and  real-time  rejection  of  EEG  artifacts  in  operational  environments. 
Future  work  includes  testing  of  the  VLSI  neuroimaging  chip  and  a  systematic  evaluation  of  the 
entire  EEG/fNIR  system  in  several  cognitive  experiments. 


TECHNOLOGY  TRANSFER 

As  mentioned  above,  we  have  initiated  collaboration  between  ARL-HRED,  ARL-NVESD  and  UCSD. 
We  plan  to  conduct  laboratory  and  field  tests  of  the  new  imaging  modality  under  the  current 
CAN-CTA. 
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